Skip to content

fix(shortlist): drop option_order when a narrow no longer fits it - #1063

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
Bruce-Yii:fix/shortlist-drop-option-order-on-narrow
Oct 10, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
Bruce-Yii:fix/shortlist-drop-option-order-on-narrow

Conversation

@Bruce-Yii

Copy link
Copy Markdown
Contributor

What

predict_shortlist narrows a choice's criteria to the top-k labels while forwarding the caller's full-length option_order, and predict_tournament builds its group and finalist questions the same way in cut(). The narrowed question then fails Agent._check_question, because the order must be a permutation of range(n_options) and the kept label set is smaller. Reproduced on main 1adc59f7:

ValueError: question 'pick': 'option_order' must be a permutation of range(2) -- one slot per option, each option once -- got [5, 4, 3, 2, 1, 0]

Fix

Following the direction you gave on #958, a narrowed choice now drops its option_order with one RuntimeWarning naming the question, the old slot count and the kept label count. The order is not remapped onto the kept labels, since that would silently move options between slots.

Both paths are covered: predict_shortlist, and cut() in predict_tournament (the round groups and the finalists share that choke point, so one helper covers them). The warning fires once per question per call, not once per tournament round.

Unchanged: passthrough questions (n <= k), non-choice questions, the caller's own dicts (both paths copy before editing), and calls without an option_order — those emit no warning and behave as before, so a call with an order answers exactly like the same call without one. Verified on real weights for both paths.

Tests

  • python tests/test_shortlist.py → 219 passed, 0 failed (on main: 209 passed, 10 failed — the 10 new checks)
  • python tests/test_hooks_api.py → 572 passed, 0 failed
  • real weights (local english checkpoint, CPU): both paths answer identically with and without the order; one warning per narrowed question; passthrough keeps its order
  • repo gates: ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 clean; python -m compileall -q laya/ tests/ exit 0
  • tests/test_mcp.py could not run in my environment: the installed mcp is 1.29.1 and the suite requires >= 2.2.0. It fails the same way on unmodified main, so this is an environment block, not a result — and MCP behaviour is unchanged, since fix(mcp): keep and validate a question's option_order #958 already makes laya_shortlist refuse a narrowed question that carries an order.

Two commits: the regression tests first (red on main, green with the fix), then the fix itself.

Closes #1062

A choice narrowed by predict_shortlist or predict_tournament kept the caller's full-length option_order, which Agent._check_question then refuses: the order must be a permutation of range(n_options) and the kept label set is smaller. Nandha called this a core contract question on NandhaKishorM#958 and asked for the order to be dropped with a warning. These checks are red on main; they pin the narrowed question validating, one RuntimeWarning naming the question, an answer identical to the no-order call, and passthrough/non-choice questions untouched.
predict_shortlist narrowed criteria to the top-k labels while forwarding the caller's full-length option_order, and predict_tournament built its group and finalist questions the same way, so a choice carrying an order crashed in the agent: the order must be a permutation of range(n_options) and the kept label set is smaller. Nandha called this a core contract question on NandhaKishorM#958 and asked for the order to be dropped with a warning, since remapping it onto the kept labels would silently move options between slots.

Both paths now drop the order on a narrowed question and emit one RuntimeWarning naming the question and the slot counts. Passthrough questions, non-choice questions and the caller's own dicts are untouched, so a call with an order answers exactly like the same call without one.
@NandhaKishorM
NandhaKishorM merged commit 6192550 into NandhaKishorM:main Oct 10, 2026
38 of 40 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged. This follows the direction I gave on #958 exactly, including the part I cared most about.

Dropping the order rather than remapping it onto the kept labels is the right call, and your reason is the right reason: remapping would silently move options between slots, so a caller who passed a deliberate order would get a different one back with no indication. A warning that names the question, the old slot count and the kept count tells them precisely what happened.

Verified on the merged tree with a 6-option choice narrowed to 2:

narrowed call succeeded: 3
warning: laya.shortlist: question 'pick': 'option_order' (6 slots) was dropped because
         the label set was narrowed to 2; ...

On main the same call raised ValueError: 'option_order' must be a permutation of range(2).

Covering cut() in predict_tournament as well as predict_shortlist is what makes this complete rather than half done. I flagged that exact interaction when merging #950: the tournament builds its group and finalist questions with dict(questions[qid], criteria=...), so it inherited the same defect, and one helper at the shared choke point covers both. Firing once per question per call rather than once per tournament round is the right granularity, since a three-round tournament would otherwise emit the same warning three times for one caller mistake.

The unchanged list is the part that makes this safe to take: passthrough questions, non-choice questions, calls without an order, and the caller's own dicts all behave exactly as before, and a call with an order now answers identically to the same call without one.

On your note about tests/test_mcp.py: you are right that it is an environment block rather than a result, and right that MCP behaviour is unchanged here, since #958 already makes laya_shortlist refuse a narrowed question carrying an order. That suite passes on the merged tree. Saying "it fails the same way on unmodified main" is the correct way to report an environment problem, and it saved me checking.

Now that core handles this, the refusal in laya_shortlist from #958 can be revisited: it exists only because core used to crash. That is a separate PR and not urgent.

All 86 suites pass, and the 82-row real-weights decision diff is unchanged.

NandhaKishorM added a commit that referenced this pull request Oct 10, 2026
14 pull requests from 8 contributors.

* the TypeScript tree parses again: an unterminated template literal shipped in
  0.4.1 and stopped 22 of 34 test files loading (#1056)
* x_jev_confidence on /v1/systemone, computed locally (#1066, #302)
* a narrowed shortlist drops option_order with a warning instead of failing (#1063)
* registering a name one edit from a built-in warns (#1055, #919)
* compose forwards four knobs the runtime already read (#1053, #1054)
* Java Router.predictBatch and Shortlist.predictTournament (#1060, #1061)
* teacher-agreement evals, with the self-agreement ceiling (#1071, #919)

Security: no malicious content found; the audit is recorded locally.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NandhaKishorM pushed a commit that referenced this pull request Oct 11, 2026
The wrapper has refused a narrowed choice with a full-length option_order
since #958, because core then raised ValueError on it. #1063 changed core
to drop the order instead of remapping it, so the refusal now blocks a
call core can answer, and deleting it without a signal would silently
lose the caller's deliberate order: MCP clients read JSON and never see
core's RuntimeWarning about the drop.

The tests demand that the narrowed call answers over the kept labels with
a per-question option_order_dropped signal naming the old and kept option
counts, visible through the JSON wrapper; no signal for passthrough,
non-choice or order-free calls; an invalid order still refused as
invalid_questions before any predict; the caller's dict unchanged and
stdout clean, so the MCP JSON-RPC channel cannot be corrupted.
NandhaKishorM pushed a commit that referenced this pull request Oct 11, 2026
#958's refusal existed only because core used to crash on a narrowed
question carrying a full-length order; #1063 made core drop the order
rather than remap it onto the kept labels, so the MCP wrapper can now
answer the call. Remove the blanket refusal -- validate_questions still
rejects anything that is not a valid permutation before predict runs --
and report the drop in the response as option_order_dropped, per
question, with the old and kept option counts. The signal is derived
from the original question and the returned shortlist meta, never from
captured warnings, which are process-global and could attribute another
call's drop to this one.

Passthrough (n <= k), non-choice and order-free responses are what they
were: the key appears only when an order was actually dropped. The tool
description, docstring and docs/cli-mcp.md name the new response key.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug]: option_order becomes invalid when shortlist or tournament narrows a choice

2 participants